The Correlator

Starting the correlator

The correlator is Apama’s core event processing and correlation engine.

The correlator executable starts the correlator. This is located in the Apama/bin directory.

To start the correlator, run the following command:

correlator [ options ]

Usually a file or directory is specified to provide YAML that configures the correlator and provides the list of files making up your Apama application, typically generated using engine_deploy. For example:

correlator --config $APAMA_WORK/my-application-deployment-dir

When you run this command with the –h option, the usage message for this command is shown.

By default, the correlator executable starts a correlator process on the current host, and configures it to listen on port 15903 for monitoring and management requests.

On start-up, the executable displays the current version number and configuration settings.

To terminate a correlator process, press Ctrl+C in the window in which it was started. Alternatively, you can issue the engine_management command with the --shutdown option. See Shutting down and managing components. Regardless of which technique you use to terminate the correlator, Apama first tries to shut down the correlator cleanly. If this is not possible, for example, perhaps because of a monitor in an infinite loop, Apama forces the correlator to shut down.

Correlator command line options

The correlator executable takes the options listed below.

Note: Most of these options can also be specified using YAML configuration, and usually YAML configuration is preferred over directly specifying these options in a script. If an option is specified in both the command line and a YAML configuration file, then the command line takes precedence. For further information, see YAML configuration file for the correlator which lists the available elements.

Option	Description
`-V \| --version`	Displays version information for the correlator.
`-h \| --help`	Displays usage information.
`-p port \| --port port`	Specifies the port on which the correlator should listen for monitoring and management requests. The default is `15903`. Alternatively, you can specify the port in a YAML configuration file. See Specifying the correlator port number for details.
`-f file \| --logfile file`	Specifies the path of the main log file that the correlator writes messages to. The default is `stdout`. See also Specifying log filenames, Correlator status log fields and Text internationalization in the logs.
`-v level \| --loglevel level` or `-v category=level \| --loglevel category=level`	Specifies the level of information that the correlator sends to the main correlator log file. Specify one of the following: - A log level which is to apply to all messages written to the log file. A category with the log level for that category. This option can be provided multiple times. You can also specify the log level in a YAML configuration file. See Setting correlator and plug-in log files and log levels in a YAML configuration file for details. This topic also lists the category names that can be specified. A log level can be one of the following (in increasing order of verbosity): `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`. The default is `INFO`. The use of `DEBUG` and `TRACE` is not recommended in a production environment as the amount of logging information will impact performance. The use of `ERROR` and `WARN` is not recommended. These log levels may make it impossible to provide support due to the lack of information in the log file. If one of these log levels is used, a warning is printed at the top of the correlator log file. Note: If `OFF`, `CRIT` or `FATAL` is specified, the log level is reset to `ERROR` and a warning message is printed at the top of the correlator log file.
`-t \| --truncate`	Specifies that if the main correlator log file already exists, the correlator should empty it first. The default is to append to it.
`-N name \| --name name`	Assigns a name to the correlator. The default is `correlator`. If you are running a lot of correlators you might find it useful to assign a name to each correlator. A name can make it easier to use the `engine_management` tool to manage correlators and adapters.
`-m num \| --maxoutstandingack num`	Specifies that you want the correlator to buffer messages for up to `num` seconds for each receiver that the correlator sends events to. The default is `10`. For additional information, see Determining whether to disconnect slow receivers.
`-M num \| --maxoutstandingkb num`	Specifies that you want the correlator to buffer the events for each receiver up to the size in kb represented by `num`.
`-x \| --qdisconnect`	Specifies that you want the correlator to disconnect receivers that are consuming events too slowly. For details, see Determining whether to disconnect slow receivers. The default is that the correlator does not disconnect slow receivers.
`--logQueueSizePeriod p`	Sets the interval at which the correlator sends information to its log file. The default is 5 seconds. Replace `p` with a `float` value for the period you want. CAUTION: Setting `logQueueSizePeriod` to `0` turns logging off. Without correlator logging information, it is impossible to effectively troubleshoot problems. See also Correlator status log fields.
`-j \| --java`	Enables support for Java applications, which is needed to inject a Java application or plug-in using `engine_inject –j`.
`-J option \| --javaopt option`	Specifies an option or property that you want the correlator to pass to the embedded JVM. You must specify the `-J` option for each JVM option. You can specify the `-J` or `--javaopt` option multiple times in the same `correlator` command line. For example: `-J "-Da=value1" -J "-Db=value2" -J "-Xmx400m"` You can use `-J` as a prefix. In this case, you have to specify it without a space: `-Joption`. For example: `-J-Dproperty=value` You can also specify JVM options in a YAML configuration file. If the same JVM option is specified in both the command line and the configuration file, the command line takes precedence. See also Specifying JVM options. Note: It is not possible to configure the correlator’s system classpath using the `CLASSPATH` environment variable. We recommend that you set the classpath on a per-plug-in basis, for example, in the descriptor file for an EPL plug-in (see Specifying the classpath in deployment descriptor files) or in the configuration file for a connectivity plug-in. For cases where you really need to set the global system classpath, you can use `-J-Djava.class.path=path`. When you use this option to pass the classpath to the JVM, Apama prepends the correlator-internal JAR files to the path you specify.
`--inputLog file`	Specifies the path of the input log file. The correlator writes only input messages to the input log file. If there is a problem with your application, Cumulocity product support can use the input log to try to diagnose the problem. An input log contains only the external inputs to the correlator. There is no information about multi-context ordering. Consequently, if there is more than one context, there is no guarantee that you can replay execution in the same order as the original execution. See Replaying an input log to diagnose problems.
`--configLog file`	Specifies the path to the configuration log file. The configuration log file contains the correlator’s configuration, that is, the contents of all YAML configuration files and properties files as well as the correlator startup arguments and environment. The configuration log is a good way to capture useful diagnostic information from the correlator’s startup in performance-critical situations where it is not acceptable to enable the input log.
`-XsetRandomSeed int`	Starts the correlator with the random seed value you specify. Specify an integer whose value is in the range of 1 to 232. The correlator uses the random seed to calculate the random numbers returned by the `integer.rand()` and `float.rand()` functions. The same random seed returns the same sequence of random numbers. This option is useful when your application uses the `integer.rand()` and `float.rand()` functions and you are using an input log to capture events and messages coming into a correlator. If you need to reproduce correlator behavior from that input log, you will want the correlator to generate the same random numbers as it generated when the original input was captured.
`--inputQueueSize int`	Sets the maximum number of spaces in every context’s input queue. The default is that each input queue has 20,000 spaces. If events are arriving on an input queue faster than the correlator can process them the input queue can fill up. You can set the `inputQueueSize` option to allow all input queues to accept more events. Typically, the default input queue size is enough so if you find that you need to increase the size of the input queue you should try to understand why the correlator cannot process the events fast enough to leave room on the default-sized queue. If you notice that adapters or applications are blocking it might be because a public context’s input queue is full. To determine if a public context’s input queue is full, use output from the `engine_inspect` tool in conjunction with the status messages in the main correlator log file.
`-g \| --nooptimize`	Disables correlator optimizations that hinder debugging. Specify this option when you plan to run the `engine_debug` tool. You cannot run the `engine_debug` tool if you did not specify the `-g` option when you started the correlator.
`--pidfile file`	Specifies the name of the file that contains the process identifier. This file is created at process startup and can be used, for example, to externally monitor or terminate the process. The correlator will remove that file after a clean shutdown. It is recommended that the file name includes the logical name of the correlator and/or port number to distinguish different correlators (for example, `my-correlator-15903.pid`). CAUTION: If the correlator process is terminated uncleanly or if the operating system is restarted, the pidfile usually remains on disk. However, the process identifier it contains is no longer valid or may match some other newly started process. To prevent mistakenly sending signals to the wrong process, ensure that the pidfile is explicitly deleted after a restart of the operating system or after an unclean termination of the correlator. Alternatively, you can configure the pidfile to be written to a transient location such as `/run` that is automatically deleted when the operating system is started. The correlator takes an exclusive lock on the pidfile while it is running. This means that if you have another correlator running using the same pidfile, the second correlator will fail to start up. You should not run multiple correlators from the same configuration using the same pidfile. In other cases, existing pidfiles will be overwritten, even if they contain a process identifier of a running process.
`--runtime mode`	On Linux 64-bit systems, you can specify whether you want the correlator to use the compiled runtime or the interpreted runtime, which is the default. Specify `--runtime compiled` or `--runtime interpreted`. The interpreted runtime compiles EPL into bytecode whereas the compiled runtime compiles EPL into native code that is directly executed by the CPU. For many applications, the compiled runtime provides significantly faster performance than the interpreted runtime. Applications that perform dense numerical calculations are most likely to have the greatest performance improvement when the correlator uses the compiled runtime. Applications that spend most of their time managing listeners and searching for matches among listeners typically show a smaller performance improvement. Using the compiled runtime requires that the binutils package is installed on the Linux system. Other than performance, the behavior of the two runtimes is the same except for the following: The interpreted runtime allows for the profiler and debugger to be switched on during the execution of EPL. The compiled runtime does not permit this. For example, you cannot switch on the profiler or debugger in the middle of a loop. The amount of stack space available is different for the two runtimes. This means that recursive functions run out of stack space at a different level of recursion on the two runtimes.
`--runtime-cache dir`	Enables caching of compiled runtime objects in the specified directory. Subsequent injections of the same files to any correlator using that cache will be quicker. For more information, see Injection time of compiled runtime.
`--frequency num`	Instructs the correlator to generate clock ticks at a frequency of `num` per second. Defaults to `10`, which means there is a clock tick every 0.1 seconds. Be aware that there is no value in increasing `num` above the rate at which your operating system can generate its own clock ticks internally. On UNIX this is usually 100.
`-Xclock \| --externalClock`	Instructs the correlator to disable its internal clock. By default, the correlator uses internally generated clock ticks to assign a timestamp to each incoming event. When you specify the `-Xclock` option, you must send time events (`&TIME`) to the correlator. These time events set the correlator’s clock. For additional information, see Determining whether to disable the correlator’s internal clock.
`--config file`	Used to configure the correlator. Specifies one of the following: - The name of a `.properties` file. See also Using properties files. The name of a `.yaml` file. See also Using YAML configuration files. The name of a directory containing `.yaml` files and `.properties` files. In this case, the files are processed in alphabetical order. A semicolon-delimited list of `.properties` files, `.yaml` files and directories. This option can be specified multiple times. The options are processed in the same sequence in which they are specified on the command line. If multiple `.yaml` files are specified, they are merged together based on the contents of the top-level maps they each contain. For example, if two files have a top-level map called `connectivityPlugins`, the merged document has a single `connectivityPlugins` map with all keys and values. Both maps, however, must not contain the same keys, otherwise an error will occur. For more information, see Configuring the correlator.
`-Dkey=value`	Specifies a value for a substitution property to be used by the Apama configuration files. The `-D` arguments always take precedence over any arguments defined in a `.properties` file. Therefore, they are processed before any `--config` arguments, regardless of the order on the command line. Using this option is similar to specifying the substitution using a `.properties` file for YAML (see also Using properties files). Note that `-D` properties are not related to Java system properties. If your intention is to set a Java system property, use `-J-Dkey=value` (and not `-Dkey=value`).
`--applicationLogLevel level`	Specifies the level of information that the correlator sends to the EPL log file. The log level must be one of the following (in increasing order of verbosity): `OFF`, `CRIT`, `FATAL`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`. The default is the current setting of the `-v` (or `--loglevel`) option. You can also specify the log level in a YAML configuration file. See Setting EPL log files and log levels in a YAML configuration file for details.
`--applicationLogFile file`	Specifies the path of the EPL log file that the correlator writes messages to. The default is the current setting of the `-f` (or `--logfile`) option. You can also specify the EPL log file in a YAML configuration file. See Setting EPL log files and log levels in a YAML configuration file for details.
`--shutdownTimeoutSecs number`	Specifies the maximum time allowed for the correlator to shut down. When this time has elapsed, the correlator shuts down forcibly, ignoring any transport or plug-in shutdown activities. This option is especially useful to prevent indefinite hangs caused by plug-ins. If this option is not provided, a default value of 90 seconds is used.
`--python=boolean`	`--python` or `--python=true` enables Python support. By default, Python support is not enabled. You can also enable Python support using a YAML configuration file. For more information, see Using Python plug-ins. `--python=false` disables Python support, overriding any value that was specified in a YAML configuration file.

Exit status

The correlator executable returns the following exit values:

Value	Description
`0`	The correlator terminated successfully.
`1`	An error occurred which caused the correlator to terminate abnormally.

Specifying log filenames

A correlator can send information to the following log files:

Main correlator log file. Upon correlator startup, the default behavior is that the correlator logs status information to stdout. To send this information to a file, specify the -f file or --logfile file option and replace file with a log filename. The format for specifying a log filename is described below.

Before you specify a log filename, you should consider your log rotation policy, which can determine what you want to specify for the log filename. See Rotating correlator log files.
EPL log files. You can create log files for packages, monitors, and events in your application. The format you use to specify these log filenames is the same as for the main correlator log file. For details about how to create EPL log files, see Setting EPL log files and log levels in a YAML configuration file and Setting EPL log files and log levels dynamically.
Correlator input log file. When you start a correlator, you can specify the --inputLog file option so that the correlator maintains a special log file for all inputs. Again, before you specify a log filename, you should consider the rotation policy for your input log files. See Rotating an input log file.

Note: The correlator input log file is slightly different from the other log files, and it is not intended to be read by a human.
Correlator configuration log file. When you start a correlator, you can specify the --configLog file option so that the correlator maintains a special log file for all of the correlator’s configuration.

The format for specifying a log filename is as follows:

file[${START_TIME}][${ROTATION_TIME}][${ID}][${PID}].log

The following table describes each part of a log filename specification. You cannot include spaces. You can separate the parts of the filename specification with underscores. You can specify ${START_TIME}, ${ROTATION_TIME}, ${ID} and/or ${PID} in any order. For examples, see Examples for specifying log filenames.

Element	Description
`file`	Replace `file` with the name of the file that you want to be the log file. If you specify the name of a file that exists, the correlator appends to it. Required. If you also specify `${START_TIME}` and/or `${ROTATION_TIME}` and/or `${ID}` and/or `${PID}`, the correlator prefixes the name you specify to the time the correlator was started and/or the time the log file was rotated (logging to a new file began) and/or an ID beginning with “001” and/or the process ID. Be sure to specify a location that allows fast access.
`${START_TIME}`	Tag that indicates that you want the correlator to insert the date and time that it started into the log filename. Optional, however you probably want to always specify either this option or `${ROTATION_TIME}` to avoid overwriting log files.
`${ROTATION_TIME}`	Tag that indicates that you want the correlator to insert the date and time that it starts sending messages to a new log file into the log filename. Optional. If you specify `${ROTATION_TIME}` and this log filename specification appears in a correlator start-up command then the name of the initial log file contains the time the correlator started.
`${ID}`	Tag that indicates that you want the correlator to insert a three-digit ID into the log filename. The ID that the correlator inserts first is “001”. Optional. The log ID increment is related only to rotation of log files. See Rotating correlator log files and Rotating an input log file. The ID allows you to create a sequence of log files. Each time the log file is rotated, the correlator increments the ID. A sequence of log files have the same name except for the ID. If you also specify `${ROTATION_TIME}` then a sequence of log files have the same name except for the rotation time and the ID. Restarting the correlator always resets the ID portion of the log filename to “001”.
`${PID}`	Tag that indicates that you want the correlator to insert the process ID into the log file name. Optional. The process ID will be constant for the lifetime of the process. Therefore, if you start multiple processes with the same arguments, they get different file names.

If you plan to rotate log files then be sure to specify ${ROTATION_TIME} or ${ID}. You can also specify both.

Examples for specifying log filenames

This topic provides examples of specifying log filenames. The format for specifying a log filename is the same in the following cases:

Starting the correlator and specifying a main correlator log file with the --logfile option.
Starting the correlator and specifying a correlator input log file with the --inputLog option.
Invoking engine_management --setLogFile to rotate the main correlator log.
Invoking engine_management --setApplicationLogFile to create an EPL log file for a package, monitor or event.

The following specifies that the name of the main log is “correlator_status.log”:

--logfile correlator_status.log

Suppose that the correlator processes events for a while, sends information to correlator_status.log, and then you find that you need to restart the correlator. If you restart the correlator and specify the exact same log filename, the correlator overwrites the first correlator_status.log file. To avoid overwriting a log, specify ${START_TIME} in the log file name specification when you start the correlator. For example:

--logfile correlator_status_${START_TIME}.log

The above command opens a log with a name something like the following:

correlator_status_2015-03-12_15:12:23.log

This ensures that the correlator does not overwrite a log file. Now suppose that you want to be able to rotate the log, so you specify the ${START_TIME} and ${ID} tags:

correlator_status_${START_TIME}_${ID}.log

The initial name of the log file is something like the one on the first line below. If you then rotate the log file then the correlator closes that file and opens a new file with a name like the one on the second line:

correlator_status_2015-03-12_15:12:23_001.log
correlator_status_2015-03-12_15:12:23_002.log

To specify an EPL log filename for messages generated in com.example.mypackge, you can specify the log filename as follows:

mypackage_${ID}_${ROTATION_TIME}.log

With that specification, messages generated in com.example.mypackage will go to a file with a name such as the one on the first line below. The time in the initial EPL log filename is the time that the initial log file is created. If you rotate the logs every 24 hours at midnight then the names of subsequent EPL log files will be something like the names in the second and third lines below.

mypackage_001_2015-03-21_18:42:06.log
mypackage_002_2015-03-22_00:00:00.log
mypackage_003_2015-03-23_00:00:00.log

If you want to run multiple correlators with the same arguments but with separate log files, you can use the process ID to differentiate them:

--logfile correlator_${PID}.log

The above command will produce a log file with a name such as the following, where each correlator will have a unique log file:

correlator_23487.log

NB: In most UNIX shells, when you start a correlator you most likely need to escape the tag symbols, like this:

correlator --inputLog input_\${START_TIME}_\${ID}.log

Correlator status log fields

The correlator sends information to the main correlator log file every five seconds (the default behavior) or at an interval that you specify with the --logQueueSizePeriod option when you start the correlator.

You will see a line starting with Correlator Status:. See Correlator status for detailed information on the log fields that are shown in this line.

Correlator status

When logging at INFO level, this information contains the following:

Correlator Status: sm=11 nctx=1 ls=60 rq=0 iq=0 oq=0 icq=0 lcn="<none>" lcq=0 lct=0.0 rx=5 tx=17 rt=31 nc=2 vm=325556 pm=81068 runq=0 si=915.3 so=0.0 srn="<none>" srq=0 jvm=0

Where the fields have the following meanings (see List of correlator status statistics for more information):

Field	Meaning	Trend
`sm`	The number of monitor instances, also known as sub-monitors.	Steady
`nctx`	The number of contexts in the correlator, including the main context.	Steady
`ls`	The number of listeners in all contexts. This includes `on` statements and active stream source templates.	Steady
`rq`	The sum of routed events on the route queues of all contexts.	Low
`iq`	The number of executors on the input queues of all contexts. As well as events, this can include clock ticks, spawns, injections and other operations. A context in an infinite loop will grow by 10 per second due to clock ticks. Every context has an input queue, which by default is a maximum of 20,000 entries.	Low
`oq`	The number of events waiting on output queues to be dispatched to any connected external consumers/receivers.	Low
`icq`	The number of events on the input queues of all public contexts. See also About context properties for information on the `receiveInput` flag.	Low
`lcn`	The name of the slowest context. This may or may not be a public context.	None
`lcq`	The number of events on the slowest context’s queue, as identified by the name of the slowest context.	Low
`lct`	For the context identified by the slowest context name, this is the time difference in seconds between its current logical time and the most recent time tick added to its input queue.	Low
`rx`	The number of events that the correlator has received from external sources since the correlator started. This includes connectivity plug-ins, `engine_send`, other correlators connected with `engine_connect`, and events that are not parsed correctly. This number excludes events sent within the correlator from EPL monitors or EPL plug-ins.	Increasing
`tx`	The number of events that have been delivered to external consumers/receivers. This counts for each external consumer/receiver an event is sent to. It counts the number of deliveries of events.	Increasing
`rt`	The number of events that have been routed across all contexts since the correlator was started.	Increasing
`nc`	The number of external consumers/receivers connected to receive emitted events. This includes connectivity plug-ins, `engine_receive`, or correlators connected using `engine_connect`.	Steady
`vm`	Virtual memory in kilobytes.	Steady
`pm`	Physical memory in kilobytes.	Steady
`runq`	The number of contexts on the run queue. These are the contexts that have work to do but are not currently running.	Low
`si`	The number of pages per second that are being read from swap space. If this is greater than zero, it may indicate that the machine is under-provisioned, which can lead to reduced performance, connection timeouts and other problems. Consider adding more memory, reducing the number of other processes running on the machine, or partitioning your Apama application across multiple machines.	Low
`so`	The number of pages per second that are being written to swap space. If this is greater than zero, it may indicate that the machine is under-provisioned, which can lead to reduced performance, connection timeouts and other problems. Consider adding more memory, reducing the number of other processes running on the machine, or partitioning your Apama application across multiple machines.	Low
`srn`	The name of the consumer/receiver with the largest number of incoming events waiting to be processed. This is the slowest non-context consumer/receiver of events, which can be an external receiver or an EPL plug-in.	None
`srq`	The number of events on the slowest consumer’s/receiver’s queue, as identified by the name of the slowest consumer/receiver.	Low
`jvm`	The sum of all memory used by the Java virtual machine (JVM) which is embedded in the correlator (that is, the used heap memory, the used non-heap memory, and the used buffer pool memory). The value is in megabytes. If the JVM is disabled, the value will be 0.	Steady

Text internationalization in the logs

The strings in the logs are encoded in two different ways, depending on whether they are written to a log file or to the console:

Log file

Apama log files are always encoded as UTF-8. Any non-ASCII strings in a log file (such as Chinese characters, German umlauts, some currency symbols, and so on) will only look correct if your text editor is configured to view the text as UTF-8. This is irrelevant on most Linux installations, where the default encoding is usually UTF-8.
Console

If the Apama component is logging to the console, the output is encoded according to the locale of your system. In practice, this means that any strings that are logged to the console (including non-ASCII strings) will look correct.

Determining whether to disconnect slow receivers

The correlator sends events to multiple receivers. Sometimes, a receiver cannot consume its events fast enough for the correlator to continue sending them. When this happens, the default behavior is that the correlator suspends processing until the receiver disconnects or starts consuming events fast enough. In other words, a slow receiver can prevent other consumers from receiving events. However, you might prefer to have the correlator disconnect a slow receiver and continue processing and sending events to other consumers. Information in this section can help you determine whether to disconnect slow receivers.

Description of slow receivers

The correlator uses two strategies to detect slow receivers: time-based, and size-based.

Time-based (maxOutstandingSecs) slow consumer detection

Events that the correlator sends to other correlators are acknowledged by the receiving correlator after the event has been delivered to the receiver. By default, if the correlator does not receive an acknowledgment within 10 seconds after the correlator sent the event, the correlator marks that receiver as being slow to consume events.

For most systems, and assuming that the underlying network connections are not prone to drop-outs, the default setting of 10 seconds is usually adequate.

You can control the length of time within which the receiver must acknowledge an event before it is marked as a slow receiver. To do so, you can specify maxOutstandingSecs in the YAML configuration file that is used when starting the correlator. See YAML configuration file for the correlator.

For example, if you specify maxOutstandingSecs: 15.0 in the YAML configuration file, the correlator marks a receiver as slow if the correlator does not receive an acknowledgment within 15 seconds. If you do not specify this element, the default is 10 seconds. You should not specify a value under 1 second because doing so raises the risk that the correlator might designate a receiver as slow when it is in fact not slow.

The mechanism that flags a receiver as slow is not precise. If a receiver does not acknowledge an event sequence after 10 seconds (the default setting), the correlator does not immediately designate the receiver as slow. Typically, the designation happens within the next 5 seconds. If you change the value of maxOutstandingSecs, the slow designation takes effect between 1 and 1.5 times the value of maxOutstandingSecs.

Size-based (maxOutstandingKb) slow consumer detection

maxOutstandingKb can also be specified in the YAML configuration file that is used when starting the correlator. See YAML configuration file for the correlator.

The correlator keeps track of the events that have been sent to each receiver but have not yet been acknowledged, based on the amount of memory taken up by those events. The correlator marks a receiver as being slow if the size of the events waiting to be acknowledged goes above the maxOutstandingKb limit, which is 10MB by default.

Note that the size-based slow consumer detection operates completely independently of the time-based (maxOutstandingSecs) slow consumer detection.

It is rare for the size-based limit to be exceeded unless the events being transmitted are very large.

Connectivity plug-ins slow consumer detection

Connectivity chains use time-based slow consumer detection similar to maxOutstandingSecs. The size of the time window is currently set at 10 seconds (not configurable). A chain that has not processed a message for more than 10 seconds is logged as slow, but is not disconnected.

How frequently slow receivers occur

In practice, sending acknowledgments should not be slow because a dedicated thread sends acknowledgments. Network interruptions are the most common cause of delayed acknowledgments. Of course, network interruptions affect events being sent as well.

Most receivers, including the engine_receive tool, normally send acknowledgments 0.1 seconds after the message was sent. Consequently, there is very little chance of a receiver being mistakenly designated as slow. In production, slow receivers should be rare as long as you have done the appropriate load testing before deployment.

If an engine library client blocks in the middle of a sendEvents call, the receiver cannot acknowledge messages while the client is blocked. As you know, a receiver is made up of an engine library and a client. Clients receive events by registering a sendEvents callback with the engine library. When the engine library gets an event from the correlator, it calls sendEvents. Problems that can cause a client to block are typically related to I/O, networking, or synchronization. The sendEvents call cannot complete until the problem is resolved. The receiver cannot send the acknowledgment until the sendEvents call completes. For example, the engine_receive tool is a receiver that is made up of an engine library and a client whose sendEvents callback writes events received to a disk file. If the client has to wait for the disk, then it is blocked. The likelihood of a sendEvents callback being blocked depends on what the client is doing. If the client is writing to a local disk, the process might block sometimes, but never more than a fraction of a second. However, sending the events over a slow or unreliable network might block for a while if the network, or the remote system cannot keep up with the event rate.

During development of event consumers, however, slow receivers are more likely. This can happen when a newly developed consumer receives an event from the correlator but cannot send the acknowledgment because of a deadlock. Another development problem might be that the event consumer cannot keep up with the load. If you have problems with slow receivers during development, you probably need to evaluate the design of your application.

Correlator behavior when there is a slow receiver

When the correlator has a slow receiver, it can behave in one of two ways:

The default behavior is that the correlator blocks further processing. This is because a slow receiver causes the correlator’s event output queue to become full. When the queue is full, the correlator stops processing because it has no place to put events. The processing thread stops and does not execute any more EPL code. The transport thread does not send any more events to any of the correlator’s other receivers. The correlator resumes processing when the slow receiver disconnects or acknowledges the outstanding sequence number.
The correlator disconnects the slow receiver, and continues processing events and sending them to its other receivers. To obtain this behavior, you specify the -x (or --qdisconnect) option when you start the correlator. The correlator sends a message to the receiver to indicate that the correlator is disconnecting the receiver. It is up to the receiver to reconnect.

To ensure that it has received an acknowledgment for every event sent, the correlator buffers each event that it sends until it receives the message’s corresponding acknowledgment. When there is a slow receiver, this can use a lot of memory if the correlator is sending a large number of messages.

Tradeoffs for disconnecting a slow receiver

When you specify the -x option when you start the correlator, it means that the correlator always disconnects a slow receiver. There are two main disadvantages to this:

The correlator loses the events that it sent to that receiver.
It is possible for the correlator to disconnect a receiver that is temporarily overloaded, and to therefore lose events unnecessarily.

Clearly, losing events can be a very serious problem. This is why the default is that the correlator does not disconnect slow receivers.

The advantage of disconnecting a slow receiver is that the correlator continues processing events.

The correlator always sends a warning message to its main log when it detects a slow receiver. This lets you see where there are potential problems.

If you cannot allow the correlator to lose events, do not specify the -x option when you start the correlator.

Determining whether to disable the correlator’s internal clock

When you start the correlator, you can specify the -Xclock option to disable the correlator’s internal clock. By default, the correlator uses internally generated clock ticks to assign a timestamp to each incoming event. When you specify the -Xclock option, you must send time events (&TIME) to the correlator. These time events set the correlator’s clock.

Use &TIME events in place of the correlator’s internal clock when you want to reproduce the historic behavior of an application.

A situation in which you might want to generate &TIME events is when you want to run experiments at faster than real time but still obtain correct timestamp behavior. In this situation, the correlator uses the externally generated time events instead of real time to invoke timers and wait events.

Disabling the correlator’s internal clock, and sending external time events, affects all temporal operations, such as timers and wait statements.

Regardless of whether the correlator generates internal clock ticks, or receives external time events, the correlator assigns a timestamp to each incoming event. The timestamp indicates the time that the event arrived on the context’s input queue. The value of the timestamp is the same as the last internally-generated clock tick or externally-generated time event. For example, suppose you have the following events and clock ticks:

&TIME(1)
A()
B()
&TIME(2)
C()

A and B receive a timestamp of 1. C receives a timestamp of 2.

Injection time of compiled runtime

Injection times for systems using the compiled runtime can be very long - significantly longer than if using only the interpreted runtime. Subsequent injection times can be improved by using the --runtime-cache dir option, which specifies a directory where the correlator can cache the compilation state (see Starting the correlator). This stores the results of compiling EPL code on disk to be used for subsequent injections of the same code.

The compiled EPL code is specific to the system on which it was compiled and the version of Apama that was used to compile it. This means that while a cache can be moved or shared between machines to improve startup on a new machine, it must be identical to the original. Otherwise, the cached version cannot be used and it must be recompiled

An injection is able to use a cached version of a previous injection if all of the following are the same as in a previous injection:

The EPL source code.
The source code of all files that contain any type that an injection depends on.
The correlator version.
The host operating system.
The CPU model.

The results of injections can be affected by any of the above. Therefore, if any change occurs, the correlator will re-compile the EPL.

The cache is designed to be used to speed up re-injection on production systems and not for quicker development cycles, which should typically use the interpreter for faster injection times. If there are identical user acceptance testing (UAT) and production environments, then the UAT environment can prime a cache which can then be used by the production correlator to improve initial startup times. However, the two systems must be identical. The strings used to disambiguate systems are logged at correlator startup when using the compiled runtime and can be used to compare the systems.

The cache contents are never removed by the correlator. If you change your source, correlator versions or platform, then the cache may grow and contain stale items which are no longer needed. If cache sizes become a problem, then we either recommend deleting the entire cache, or just the cache files with the oldest timestamps. The correlator will transparently recompile any needed files which are missing from the cache.

Configuring the correlator

You can configure the correlator using YAML configuration files and properties files. See the topics below for detailed information on these types of files, and on how to configure certain aspects of the correlator.

For information about configuring the files making up the Apama application the correlator should run, see Deploying Apama applications with a YAML configuration file. YAML configuration is also used to configure connectivity plug-ins as described in Configuration file for connectivity plug-ins.

Using YAML configuration files

You can specify one or more YAML files using the --config option when you start the correlator. See Starting the correlator.

For detailed information on YAML, see https://yaml.org/spec/1.2/. A quick overview is given below.

YAML configuration files can contain maps, lists or simple values:

A map contains a string key, followed by a colon and space, followed by a value (which can be a map, a list or a simple value). Typically, an entry with a simple value is written on one line, and collections (maps and lists) are written on a following line with indentation.
A list contains a number of values, each of which is written on a new line with a preceding dash and space ("- “).
A simple value includes a string or number. It is typically written on a single line. A string may be enclosed in quotes if needed, but this is not mandatory.

Some characters in YAML have special significance at certain positions. For example, a value ending with a colon (:) is treated as a key in a dictionary, so if you want a string value to end with a literal colon (:), you should quote it.
Nesting is expressed using spaces to indent different levels of object. Tabs are forbidden in YAML files, all indentation must be performed using spaces.

For example:

myTopLevelMap:
  mykey: myvalue
  mylist:
    - item 1 # comment
    - item 2
    - "a quoted string value"

YAML is a superset of JSON. Thus, any valid JSON is also usable in the YAML configuration file. This is helpful if there is ambiguity in the way YAML expresses configuration.

YAML documents should be saved with the standard UTF-8 character encoding.

YAML configuration file for the correlator

The following sample shows the format of a YAML configuration file for the correlator.

correlator:
  javaApplicationSupport: false
  pythonSupport: true
  randomSeed: number
  initialization:
    list: [... ]
  runtime: interpreted
  runtimeCacheDir: path
  inputLogFile: path
  externalClock: false
  timerFrequency: 10
  truncateLogFile: false
  disableOptimizations: false
  statusLogIntervalSecs: 5.0
  inputQueueSize: 20000
  logFile: path
  shutdownTimeoutSecs: number

server:
  pidFile: path
  port: 15903
  name: string
  bindAddress: [...]
  permittedRootURIs: [...]
  allowClient: [...]
  maxOutstandingSecs: 10.0
  maxOutstandingKb: 10240
  disconnectSlowConsumers: false

jvmOptions: [...]

eplLogging:
.root:
    level: loglevel
    file: path
  string:
    level: loglevel
    file: path
correlatorLogging:
  string:
    level: loglevel

environment:
  string: string

engineConnect:
  - sourceHost: string
    sourcePort: integer
    channels:
      - string
    mode: string
    disconnectIfSlow: boolean

includes: [...]

# For configuring EPL plug-ins:
eplPlugins: [...]

# For configuring connectivity plug-ins:
connectivityPlugins: [...]
startChains: [...]
dynamicChainManagers: [...]
dynamicChains: [...]

For paths, we recommend using replacements like ${PARENT_DIR} or ${APAMA_WORK} to make the configuration files portable. See also the list of predefined properties in Using properties files.

[...] denotes that this can be a list.

You can use multi-line and single-line syntax for maps (for example, in the eplLogging case).

Many of the elements in the above sample configuration file correspond to command-line options of the correlator executable and have the same syntax and options. See the tables below, and see the descriptions of these command-line options in Starting the correlator for detailed information.

Note:

If an element in a YAML configuration file is defined more than once, the first definition that is loaded by the correlator takes precedence over all definitions that are loaded later. An INFO message is then logged for all later definitions that are being ignored. Any corresponding options that are specified directly on the command line, however, take precedence over any other later definitions.

The following table lists the elements that can be specified in the correlator section of the YAML configuration file:

This element	corresponds to this command-line option
`javaApplicationSupport`	`-j \| --java`
`pythonSupport`	`--python`
`randomSeed`	`-XsetRandomSeed`
`runtime`	`--runtime`
`runtimeCacheDir`	`--runtime-cache`
`inputLogFile`	`--inputLog`
`externalClock`	`-Xclock \| --externalClock`
`timerFrequency`	`--frequency`
`truncateLogFile`	`-t \| --truncate`
`disableOptimizations`	`-g \| --nooptimize`
`statusLogIntervalSecs`	`--logQueueSizePeriod`
`inputQueueSize`	`--inputQueueSize`
`logFile`	`-f \| --logfile`
`shutdownTimeoutSecs`	`--shutdownTimeoutSecs`

The following table lists the elements that can be specified in the server section of the YAML configuration file:

This element	corresponds to this command-line option
`pidFile`	`--pidfile`
`port`	`-p \| --port`
`name`	`-N \| --name`
`maxOutstandingSecs`	`-m \| --maxoutstandingack`
`maxOutstandingKb`	`-M \| --maxoutstandingkb`
`disconnectSlowConsumers`	`-x \| -- qdisconnect`

The following table lists the remaining sections that can be specified in the YAML configuration file:

This element	corresponds to this command-line option
`jvmOptions`	`-J \| --javaopt`
`eplLogging/.root/level`	`--applicationLogLevel`
`eplLogging/.root/file`	`--applicationLogFile`
`correlatorLogging/level`	`-v \| --loglevel`

Other elements are described in topics under Configuring the correlator, and this also includes additional information for some of the elements that correspond to the command-line options. These are:

In the correlator section:
- initialization. See Deploying Apama applications with a YAML configuration file.
In the server section:
- bindAddress and permittedRootURIs. See Binding server components to particular addresses.
- allowClient. See Ensuring that client connections are from particular addresses.
jvmOptions. See Specifying JVM options.
eplLogging. See Setting EPL log files and log levels in a YAML configuration file.
correlatorLogging. See Setting correlator and plug-in log files and log levels in a YAML configuration file.
environment. See Setting environment variables for Apama components.
engineConnect. See Setting up connections between correlators in a YAML configuration file.
includes. See Including YAML configuration files inside another YAML configuration file.

The eplPlugins element is used for configuring EPL plug-ins. There is no equivalent command-line option. See Writing EPL plug-ins in Python for detailed information.

The following elements are used for configuring connectivity plug-ins. There are no equivalent command-line options. See Using connectivity plug-ins for detailed information.

connectivityPlugins
startChains
dynamicChainManagers
dynamicChains

Including YAML configuration files inside another YAML configuration file

Instead of specifying a list of configuration files on the command line with the --config option, you can also specify these files in the includes section of another configuration file. For example:

includes:
   - myCodec.yaml
   - myTransport.yaml

When the configuration file that is specified with the --config option is processed, the list in the includes section is expanded recursively, and the information inside the included configuration files is merged into the main configuration. The same rules are used as for the --config option; for details, see the description of that option in Starting the correlator.

You do not need to worry about multiple file inclusions (such as cyclical or diamond references), but you must still be careful not to have duplicate keys in top-level maps.

Using an includes section can be useful for specifying a connectivity chain in a modular way. For example, you may have a main configuration file with the following contents:

startChains:
  myChain:
     - apama.eventMap
     - MyCodec
     - MyTransport
includes:
   - myCodec.yaml
   - myTransport.yaml

where the included files have the following contents:

myCodec.yaml:
```
connectivityPlugins:
  MyCodec:
    classpath: ${myJarVersion}
    class: com.example.my.Codec
includes:
   - propDir
```
You can include further files in an included file. Instead of a file name, you can also specify the name of a directory in the includes section. All files that can be found in this directory are then included.

The includes section in the above example assumes that there is a directory named propDir which contains a jarVersions.properties file with the following content:
```
myJarVersion=myJar316.jar
```

myTransport.yaml:

connectivityPlugins:
  MyTransport:
    classpath: myOther.jar
    class: com.example.my.Transport

Properties defined in included files are only valid for later files. For example, the deployment with the following files will work:

1.yaml:

includes:
   - my.properties
   - 2.yaml

my.properties:
```
3=3.yaml
```
2.yaml:
```
includes:
   - ${3}
```

The deployment with the following file, however, will not work:

1.yaml:
```
includes:
   - my.properties
   - 2.yaml
   - ${3}
```
In this case, you should specify the my.properties file as an argument with the --config option, in addition to the 1.yaml file.

See Using properties files for more information.

Using properties files

Properties files (with the file extension .properties) can be used to either specify values for substitution variables in YAML files (see also Using YAML configuration files) or to configure EPL. A properties file must be either in ISO-8859-1 encoding or in UTF-8 encoding if it begins with an UTF-8 byte order mark (BOM).

You can specify one or more properties files using the --config option when you start the correlator. See Starting the correlator. The properties files are processed in the order in which they appear on the command line. Each properties file can refer to properties that have already been defined by a previously processed properties file, using ${varname} syntax.

Note: If the same property is defined more than once, the first definition that is loaded by the correlator takes precedence over all definitions that are loaded later. Messages are then logged for all later definitions that are being ignored.

The properties file format is the same as the standard Java .properties file format, with # for comments and \\ used to escape any occurrences of \. For example:

# my comment line
myplugin.mykey=foo\\bar

Properties files are by default parsed using the ISO-8859-1 encoding (Latin-1). To use characters from other character sets, you have two options. You can use a \uXXXX escape sequence, which uses the UCS-2 character codes. This allows you to encode any character from the Basic Multilingual Plane (BMP). However, we do not support UTF-16 surrogate pairs to access characters outside the BMP. Alternatively, you can provide a UTF-8 BOM as the first characters of the file. In this case, you can use any Unicode sequence in UTF-8 directly in the file without any escaping.

You can use the following predefined properties:

Property	Description
`${PARENT_DIR}`	The absolute normalized path of the directory containing the properties file or YAML file currently being processed.
`${APAMA_HOME}`	The path to the Apama installation.
`${APAMA_WORK}`	The path to the Apama work directory.
`${$}`	The literal $ sign.

All properties are applied to all YAML files, although conventionally, there is often a .properties file named the same as each .yaml file. Properties files, however, are not actually tied in to YAML files in any way. It is therefore recommended that you prefix each property key with a unique string, such as the identifier of the chain to which it applies.

Instead of using a properties file or in addition to using a properties file, you can also use the -D option of the correlator executable. See Starting the correlator.

Properties defined either through properties files or on the command line are available to EPL via the Management interface. For more details, see Using the Management interface and the API reference for EPL (ApamaDoc).

Runtime parameterization of configuration

Some components allow parameterization of configuration at runtime, for example, connectivity plug-in chains created from a manager or from EPL. Variables which must be replaced at runtime are denoted with @{varname} instead of ${varname}. Strings using this syntax may need to be escaped with double quotes in the YAML configuration file. Instead of being supplied via the command line or via properties files, these substitutions are provided within the subsystem using that part of the configuration. See also Creating dynamic chains from EPL.

Specifying the correlator port number

You can specify the port on which the correlator should listen for monitoring and management requests. To do so, you specify a port definition in the server section of the YAML configuration file. For example:

server:
  port: 15903

This is identical to specifying the --port option when starting the correlator. See also Starting the correlator.

If the port is specified with the --port option when starting the correlator, this value is used. Else, if it is specified in the YAML configuration file, that value is used. If a port is not specified at all, the default value 15903 is used.

Binding server components to particular addresses

To bind Apama server components to a particular address or set of addresses, specify a bindAddress definition for each address. Specify this in the server section of the YAML configuration file. For example:

server:
  bindAddress:
    - 127.0.0.1:15903
    - 10.0.0.1

You can specify as many bindAddress definitions as you want. Clients can connect to any of the listed addresses.

An IP address is required. If you do not specify a port, the Apama server components use the port that is specified when the correlator is started. This makes it possible to share YAML configuration files if you want to restrict connections according to only addresses.

If you do not specify a YAML configuration file when you start the correlator, or there are no bindAddress definitions in the YAML configuration file, the Apama components bind to the wildcard address (0.0.0.0).

Using the correlator web interface with non-default addresses

The correlator web interface sometimes uses the client’s host header for redirects. For security, this is checked against a list of accepted host:port aliases for this component. By default, all of the addresses of the machine on the default or overridden correlator port are part of this list.

If the correlator has a custom bind address configuration, or is being hidden behind another component which is redirecting the web interface, the default list of permitted addresses may need to be overridden. This can be done with the permittedRootURIs configuration entry:

server:
  permittedRootURIs:
     - ${EXTERNAL_HOSTNAME}:${EXTERNAL_PORT}

This can either be a single string or a list of multiple permitted URIs. This list replaces the automatically calculated defaults, it does not append.

Ensuring that client connections are from particular addresses

To ensure that client connections are from particular addresses, add one or more allowClient definitions to the YAML configuration file in the server section. For example:

server:
  allowClient:
    - 127.0.0.1
    - 192.168.128.0/17

An allowClient definition takes an IP address, as in the first example above, or a CIDR (Classless Inter-Domain Routing) address range, as in the second example above. With these example entries in the YAML configuration file, the Apama components allow connections from either the localhost (127.0.0.1) or IP addresses where the first 17 bits match the first 17 bits of 192.168.128.0. The Apama components do not accept connections from any other IP addresses.

If you specify a YAML configuration file when you start the correlator, and if there are any allowClient definitions in the YAML configuration file, then the Apama components do not allow connections from any IP address that does not fall within one of the allowClient ranges specified. If you do not specify a YAML configuration file when you start the correlator, or there are no allowClient definitions in a YAML configuration file that you do specify, the Apama components accept connections from any client.

Important

This feature is intended to prevent mistakenly connecting to the wrong server. It is not intended to prevent malicious intruders since it provides no protection against address spoofing.

Setting environment variables for Apama components

You can use the YAML configuration file to set environment variables. Put environment variable declarations in the environment section. For example:

environment:
  MY_ENV_VAR: myvalue

If you specify a YAML configuration file when you start the correlator, and if there are any environment variable entries in the YAML configuration file, then the Apama components use those environment variable settings. If you do not specify a YAML configuration file when you start the correlator, or there are no environment variable entries in a YAML configuration file that you do specify, the Apama components use the environment variable settings specified elsewhere.

Note:

You cannot use this feature to set variables such as LD_PRELOAD and LD_LIBRARY_PATH because UNIX dictates that they are set before the affected process starts execution. These environment variables should therefore be considered read-only.

Setting EPL log files and log levels in a YAML configuration file

This topic describes how to configure logging for individual EPL packages. For information about configuring the log level of the whole correlator and plug-ins running inside it, see Setting correlator and plug-in log files and log levels in a YAML configuration file.

You can configure per-package logging in several ways:

Statically, in the YAML configuration file when starting the correlator, as described in this topic.
Dynamically, using the following options of the engine_management tool:
- --setApplicationLogFile
- --setApplicationLogLevel See Setting EPL log files and log levels dynamically for detailed information.
Dynamically from within EPL. See Using the Management interface for detailed information.

To set log files and/or log levels for EPL root, packages, monitors, or events, specify entries in the eplLogging section of the YAML configuration file.

To set the default log file and level for the EPL root package, specify this in the following format:

eplLogging:
.root:
     file: rootLogFilename
     level: ROOTLOGLEVEL

Replace rootLogFilename with the name of the log file for the EPL root package. No spaces are allowed in the log file name. Replace ROOTLOGLEVEL with TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT or OFF. For example:

eplLogging:
.root:
     file: apama/root.log
     level: ERROR

You do not need to specify both a log file and a log level; you can specify one or the other. If you do not specify a log file or log level for the root package, it defaults to the correlator’s log file/log level.

To set the default log file and level for an EPL package, monitor or event, specify this in the following format:

eplLogging:
  node:
     file: nodeLogFilename
     level: NODELOGLEVEL

Replace node with the name of the EPL package, monitor, or event for which you are setting the logging attribute. If a monitor or event is in a named package and not the root package, be sure to specify the fully qualified name. Replace nodeLogFilename with the name of the default log file for the specified EPL package, monitor or event. No spaces are allowed in the log file name. Replace NODELOGLEVEL with TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT or OFF. This is the default log level for the specified node. For example:

eplLogging:
  com.myCompany.Client:
    file: apama/Client.log
    level: DEBUG
  com.myCompany.Internal: { level: ERROR }

The above example shows both multi-line and single-line syntax. The single-line syntax is more compact when you are just setting either the log level or the file, but not both.

For a particular node, you do not need to specify both a log file and a log level; you can specify one or the other. If you do not specify a log file or log level for a particular node, it defaults to the settings for a parent node. See Tree structure of packages, monitors, and events.

When you set log attributes in the YAML configuration file, the rules for hierarchical logging apply. See Setting EPL log files and log levels dynamically.

If you pass a YAML configuration file to a correlator when you start that correlator and the configuration file contains an eplLogging section, the correlator uses the logging settings in that section. If you do not pass a configuration file when you start a correlator, or there are no settings in the eplLogging section, then correlator initialization does not include any log settings except for the default correlator log.

Whether or not you specify a YAML configuration file when you start a correlator, any log settings you specify can be overwritten after initialization by invoking the engine_management tool and specifying the --setApplicationLogFile and/or --setApplicationLogLevel options. See Managing EPL log levels and Managing EPL log files.

Note:

Regularly rotating log files and storing the old ones in a secure location may be important as part of your personal data protection policy. For more information, see Recommendations for logging by Apama application code.

Setting correlator and plug-in log files and log levels in a YAML configuration file

In a YAML configuration file, you can configure the log file and/or log level individually for plug-ins, as well as for components of the correlator itself. You can do this either for the whole correlator or for individual categories. This includes setting the log level for individual connectivity plug-ins and EPL plug-ins.

Note:

You can also set the log levels individually on the command line (see Starting the correlator). If a log level is specified on the command line, it overrides any setting in the YAML configuration file.

To set the log levels in the YAML configuration file, specify entries in the correlatorLogging section. The syntax for this section is:

correlatorLogging:
.root:
      level: ROOTLOGLEVEL
      file: ROOTLOGFILE
   category: CATEGORYLOGLEVEL

You can specify either the log level directly, or as a map with the key level. These are synonymous. To set the log file, you must use the map syntax.

.root is used to specify the default log file/level for the whole correlator. We do not recommend specifying a log level higher than INFO for the default log level, since important information may be lost from the log files.

Valid log levels are TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT or OFF.

You can use replacement tokens in a log file. See Specifying log filenames for more information.

The categories for which the logging can be configured are listed in the table below. Note that log categories are regarded as a hierarchy. For example, setting the log level for apama will be inherited by apama.status.

Example - the default log level for the whole correlator is set to INFO, and the log level for the connectivity plug-ins framework is set to DEBUG:

correlatorLogging:
.root:
      level: INFO
   apama.connectivity: DEBUG

This category	controls the following
`apama.applicationEvents`	Correlator application event logging. Provides detailed output of the inner workings of the correlator, such as context-state changes, event triggering, spawning, routing, etc. To enable application event logging, you have to set the log level of this category to `DEBUG`. See also Viewing garbage collection and application events information.
`apama.connectivity`	Connectivity plug-ins - framework.
`apama.debughandler`	Correlator EPL debugger.
`apama.messaging`	Internal messaging-related messages.
`apama.socket`	Socket-level communications.
`apama.status`	Correlator status lines.
`apama.streams`	Stream queries within EPL.
`apama.verboseGC`	Correlator garbage collection messages for all monitors. To enable garbage collection logging, you have to set the log level of this category to `DEBUG`. See also Viewing garbage collection and application events information.
`apama.verboseGC.MonitorName`	Correlator garbage collection messages for the specified monitor. To enable garbage collection logging, you have to set the log level of this category to `DEBUG`. See also Viewing garbage collection and application events information.
`com.softwareag.connectivity`	Connectivity plug-ins - Java framework.
`connectivity.PluginName.ChainName`	Connectivity plug-ins - messages from a specific plug-in. Applies to connectivity plug-ins written in both C++ and Java.
`connectivity.TransportName.ManagerName`	Connectivity plug-ins - messages from managers.
`connectivity.apama.hostPluginName.ChainName`	Connectivity plug-ins - messages from the specified host plug-in. The valid host plug-in names (such as `eventMap`) are listed in Host plug-ins and configuration.
`connectivity.chain.ChainName`	Connectivity plug-ins - chain-related messages.
`plugins.PluginName`	EPL plug-ins - in C++, Java or Python. Note that handling of Java logging is slightly different to EPL and Python. By convention, we recommend Java EPL plug-ins should specify `plugins.PluginName` when creating the `Logger` object, for example, `private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger("plugins.MyPlugin")`. However, this is only a convention, and if some other string or a Java class is specified instead, then that will be used as the correlator’s log category. For information about how to configure the log level for each category, see Setting correlator and plug-in log files and log levels in a YAML configuration file. Keep in mind that the logging for the EPL plug-ins is about logging from Java, Python and C++ plug-ins that can be called from EPL. For information on logging from EPL itself, see Setting EPL log files and log levels in a YAML configuration file.

Note:

It is not possible to dynamically change the correlator and plug-in log levels. Only EPL log levels can be dynamically changed (see also Setting EPL log files and log levels dynamically).

Setting up connections between correlators in a YAML configuration file

To configure correlators to automatically connect to other correlators and receive events from them, the YAML configuration has a section called engineConnect.

For example:

engineConnect:
  - sourceHost: localhost
    sourcePort: ${FIRST_PORT}
    channels:
      - myChannel
    mode: parallel
    disconnectIfSlow: false
  - sourceHost: localhost
    sourcePort: ${SECOND_PORT}
    channels:
      - myChannel
    mode: parallel
    disconnectIfSlow: true

The above elements correspond to command-line options of the engine_connect tool and have the same syntax and options. See the descriptions of these command-line options in Configuring pipelining with engine_connect for detailed information, but keep in mind that only the options shown in the above example are supported.

On startup, the configuration specified in the engineConnect section always uses the current correlator as the target correlator. Unlike the engine_connect tool, you cannot explicitly specify an arbitrary target correlator with engineConnect.

Unlike the engine_connect tool, when the mode is not specified for an entry in the engineConnect YAML, the default mode is parallel, meaning that there is one connection for each specified channel.

Specifying JVM options

In a YAML configuration file, you can specify JVM options which the correlator is to pass to the embedded JVM. To do so, you provide a list of JVM options with the jvmOptions key. If an option has a leading hyphen, you have to enclose the option in quotes. But you can also specify the option without quotes by leaving out the leading hyphen; in this case, the correlator will automatically add the hyphen. Thus, you can specify the JVM options as shown in the following example:

jvmOptions:
 - "-Dkey1=value1"
 - Dkey2=value2
 - "-Xms100m"
 - Xmx500m

You can specify JVM options in multiple configuration files. The options from all the files are then appended together in the order in which they have been specified. If the same JVM option is specified in the command line as well as in a configuration file, the command line takes precedence. For more details, see the description of the -J option in Starting the correlator.

Deploying Apama applications with a YAML configuration file

The main way to inject your Apama application and initial events into the correlator is to configure the list of files to be injected using a YAML configuration file that the correlator reads during startup. Typically this configuration file is created using the engine_deploy tool.

The YAML configuration file for the correlator is specified using the --config option when starting the correlator (see also Starting the correlator). The YAML file itself contains the following:

correlator:
    initialization:
        list:
            - ${PARENT_DIR}/bin/myPlugin.jar
            - ${PARENT_DIR}/eventdefinitions/evtdef.mon
            - ${APAMA_HOME}/monitors/ConnectivityPlugins.mon
            - ${PARENT_DIR}/monitors/app.mon
            - ${PARENT_DIR}/events/start.evt
        encoding: UTF8

It is recommended to use a substitution variable such as ${APAMA_HOME} or ${PARENT_DIR} rather than absolute or relative paths. This makes the configuration independent of the correlator’s current working directory.

The list entries must have one of the following extensions:

.mon for EPL monitor, aggregate and event definition source.
.jar for EPL plug-ins written in Java.
.cdp for correlator deployment packages.
.evt for event files.

Note:

If you use the engine_deploy tool, EPL injection order is automatically generated. For further information, see Deploying a correlator.

The encoding entry is optional. If UTF8 is specified, all of the text input files (.mon, .evt) are read as UTF-8. If local is specified or if the encoding entry is not specified at all, the text files are assumed to be in the local encoding unless they start with a UTF-8 byte order mark (BOM) in which case they are treated as UTF-8.

This mechanism separates the build-time (calculating injection order, generating EPL) steps from the deployment-time steps, so no build steps are required in the environment where the correlator is running. This does mean that any changes to the project potentially require rewriting the YAML list and then redeploying, however, it allows separation of these concerns.

Note:

The correlator port is opened before the injections have completed. This allows monitoring tools to connect while the injections occur, but this also means that the correlator may not be entirely ready when a client connects. You can use the flushAllQueues request (see Shutting down and managing components) to wait for the injections to complete.